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Disclaimer 


The  views  expressed  in  this  Project  Demonstrating  Excellence  are  those  of  the  author 
and  do  not  reflect  the  official  policy  or  position  of  the  Department  of  Defense  or  the  U.S. 
Government.  The  Project  Demonstrating  Excellence  documentation  is  prepared  in  the 
sponsoring  U.S.  Air  Force  Wright  Laboratory  Technical  Report  format.  The  preceding 
Standard  Form  298  on  page  2  contains  an  abstract  limited  to  a  two  hundred  (200)  word 
maximum.  This  form  is  required  for  public  release  purposes  by  the  U.S.  Air  Force  Public 
Affairs  Office  as  a  result  the  Project  Demonstrating  Excellence  sponsorship  by  the  U.S. 
Air  Force  Wright  Laboratory.  The  following  Project  Demonstrating  Excellence  Abstract 
on  page  4  is  provided  to  meet  The  Union  Institute  requirements  in  partial  fulfillment  of 
The  Doctor  of  Philosophy  Degree  in  Engineering  Management. 
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Project  Demonstrating  Excellence 
Abstract 


Program  managers  use  metrics  to  assist  their  understanding  of  progress  and 
accomplishments.  Often  the  complexities  of  large  programs  result  in  exhaustive  efforts  to 
develop  the  best  metric  which  provides  upper  management  with  a  simple  to  understand 
measure  of  goodness  or  success.  Too  often  the  tasking  to  define  a  specific  program 
metric  is  constrained  by  a  limitation  requiring  any  data  collection  to  be  minimally  intrusive 
upon  the  work  force  or  customer  base.  Persistence  and  hard  work  is  sometimes  rewarded 
with  a  validated  metric  that  management  and  stakeholders  accept.  However,  the  usual 
result  is  a  quantity  which  is  misunderstood  or  simply  not  appreciated.  The  difficulty  arises 
when  management,  holding*the  coveted  metric  in  hand,  realizes  it  has  at  best  a  good 
statement  of  past  history  but  what  it  really  needs  is  a  methodology,  an  actual  scientific 
method,  for  managing  the  future. 

This  study  addresses  the  goal  of  managing  the  future  by  combining  a  promising  metric, 
strategic  planning,  probability  encoding,  risk  management,  and  the  Taguchi  Design  of 
Experiments  technique  into  a  new  “out  of  the  box”  methodology  for  program 
management.  This  management  methodology  demonstrates  an  interdisciplinary  nature  in 
that  human  factors  and  statistical  processes  combine  in  the  identified  components  to 
produce  a  quantified  measure  predicting  future  results.  Although  the  methodology  is 
applicable  to  many  large  programs,  the  United  States  military  technology  transfer  program 
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partnering  Department  of  Defense  laboratories  with  civilian  industries  was  chosen  as  the 
focus  for  the  demonstration.  The  method  demonstrated  goes  beyond  current  documented 
attempts  at  metric  development  within  the  federal  program.  The  result  of  the  research 
yields  useful  management  information  for  the  technology  transfer  activity  of  the  Air  Force 
Wright  Laboratory.  Transferring  military  technology  to  civilian  industry  results  in 
products  and  services  solving  deficiencies  in  food  supply,  shelter,  education,  health  care, 
transportation  and  recreation  while  simultaneously  contributing  toward  the  attaintment  of 
national  employment  goals  yet  assuring  new  capabilities  for  civil  security  and  national 
defense. 

The  Wright  Laboratory  investment  in  mission  related  research  and  development 
programs  which  also  show  promise  for  potential  future  technology  transfer  will  benefit 
from  informed  management  making  good  decisions.  The  ability  to  predict  the  necessary 
investment  funds  to  accomplish  Cooperative  Research  and  Development  Agreements 
through  a  method  capable  of  measurement  by  a  proposed  metric  results  from  this  Program 
Demonstrating  Excellence.  This  new  theoretical  approach  toward  management  of  federal 
laboratory  research  and  development  programs  results  in  a  new  methodology  grounded  in 
the  theory  of  risk  analysis  capable  of  addressing  uncertainty  as  found  in  technology 
transfer. 
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Glossary 


AFIT  -  Air  Force  Institute  of  Technology  Graduate  School  of  the  Air  Force  located  at 
Wright  Patterson  Air  Force  Base,  OH. 

AFMC  -  Air  Force  Materiel  Command 

AGATE  -  Advanced  General  Aviation  Transport  Experiments 

Aggregate  Demand  -  Aggregate  demand  is  the  total  demand  for  goods  and  services  in  the 
economy.  This  demand  is  generated  by  expenditures  on  domestically  produced  goods  and 
services  by  consumers,  firms  (through  investments),  the  government,  and  by  entities  in 
foreign  countries.1 

ARR  -  Actual  Rate  of  Return 

ASC  -  Aeronautical  Systems  Center 

ASTARS  -  A  Science  &  Technology  Action  Reporting  System 

CRDA  -  Cooperative  Research  and  Development  Agreement 
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DoD  -  Department  of  Defense 


DoE  -  Department  of  Energy 


DOE  -  Design  of  Experiments 


DoT  -  Department  of  Transportation 

Internship  -  A  new  learning  experience  involving  a  significant  portion  of  time  directed 
toward  professional  and  personal  development.2 


DSMC  -  Defense  System  Management  College 


FLC  -  Federal  Laboratory  Consortium 


IE  -  Information  Economics 


HR  -  Internal  Rate  of  Return 


IT  -  Information  Technology 


IPT  -  Integrated  Product  Team 
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Learning  Agreement  -  The  signed  document  representing  a  comprehensive  plan  designed 
by  the  learner  through  collaboration  with  the  doctoral  committee.  This  plan,  as 
individualized  by  the  learner,  may  include  use  of  university  courses,  libraries,  museums, 
private  resources,  professional  associations,  communication  media,  planned  travel, 
conferences,  workshops,  lectures,  experiences  within  professional  work  settings, 
constructive  social  action  and  any  other  services  or  materials  needed  and  available  may 
become  part  of  the  strategy  for  academic  and  personal  development  toward  the  doctoral 
degree.3 

Management  of  Technology  -  The  interdisciplinary  combination  of  technical  and 
management  knowledge  focused  toward  responsibility  in  technical  problem  solving  of 
complex  demands  resulting-ffom  the  social,  technical,  economic,  and  political  facets  of 
science  and  engineering  organizations. 

Metric  -  A  measurement,  taken  over  time,  that  communicates  vital  information  about  a 
process  or  activity.  A  metric  should  drive  appropriate  action  and  must  be  linked  to  a 
strategic  plan. 

NPV  -  Net  Present  Value 

ORGA  -  Out  Reach  to  General  Aviation 
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ORTA  -  Office  of  Research  and  Technology  Applications4 

PDE  -  Project  Demonstrating  Excellence 

PE  -  Probability  Encoding 

QRA  -  Quantified  Risk  Analysis 

raNPV  -  risk  adjusted  NPV 

Risk  -  The  probability  of  an  undesirable  event  occurring  and  the  significance  of  the 
consequence  of  the  occurrence.5 

ROI  -  Return  on  Investment 

Stakeholder  -  A  person  or  legal  entity  which  shares  ownership  in  the  development, 
execution,  liability  and  responsibility  of  a  concept,  idea,  invention,  product,  or  process 
from  the  initial  developer  to  the  customer  as  end  user. 

Technological  Progress  -  Technological  progress  refers  to  increases  in  total  output  per 
unit  of  input.  Technological  progress  can  take  the  form  of:  a)  the  same  amount  of  inputs 
producing  more  valuable  products,  b)  increases  in  the  quality  of  inputs  allowing  the  same 
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amount  of  inputs  to  produce  more  output,  and  c)  the  old  level  of  output  produced  with 
fewer  inputs.6 

Technology  Transfer  -  A  process  by  which  facilities,  equipment,  or  other  resources 
relating  to  scientific  or  technological  developments  of  a  federal  laboratory  are  provided  or 
disclosed  by  any  means  to  another  industrial  organization,  including  a  corporation, 
partnership,  limited  partnership,  or  industrial  development  organization;  public  or  private 
foundation;  nonprofit  organization,  including  a  university,  or  other  person  to  enhance  or 
promote  technological  or  industrial  innovation  for  a  commercial  or  public  purpose.7 

Technology  Transition  -  The  movement  of  technology  from  R&D  to  first-time 
application.8 

TTO  -  Technical  Transfer  Office 

WL  -  Wright  Laboratory 

XP  -  Plans  Directorate 

XPT  -  Technology  Transfer  Division 


16 


1.0  Introduction 


1.1  The  Project  Demonstrating  Excellence  (PDE)  in  partial  fulfillment  of  requirements 
for  The  Doctor  of  Philosophy  Degree  in  Engineering  Management  at  The  Union  Institute 
focuses  upon  the  development  of  management  methodology  for  technology  transfer 
between  the  Air  Force  Wright  Laboratory  (WL)  and  industry  partners.  Effective 
managftTWPint  methods  will  enhance  the  national  impact  achieved  from  the  conversion  of 
former  defense  weapon  system  technology  into  new  products  for  the  civilian  industrial 
market  through  the  process  of  technology  transfer. 

1 .2  Successful  management  of  technology  transfer  processes  depends  upon  both  effective 
metrics  which  measure  the  cost  of  the  conversion  process  from  research  to  manufacturing 
and  prediction  methods  assessing  the  risk  of  new  technology  for  future  transfers.  The 
transfer  of  technology  from  military  defense  laboratories  has  potential  to  create  products 
and  services  solving  deficiencies  in  food  supply,  shelter,  education,  health  care, 
transportation  and  recreation  while  simultaneously  contributing  toward  the  attaintment  of 
national  employment  goals  and  insuring  new  capabilities  for  civil  security  and  national 
defense.  Within  the  civilian  industry  of  our  nation,  the  activity  cost  of  moving  technology 
(e.g.  technology  transfer)  into  manufactured  goods  has  traditionally,  altlhough  not  in 
every  case,  been  captured  within  overhead  cost  accounts.  Military  research  laboratories 
have  only  recently  begun  to  capitalize  upon  the  opportunity  to  aggressively  move  research 
products  into  non-mission  arenas  of  the  civilian  industry  for  mutual  economic  benefit.  The 


17 


U.S.  Department  of  Defense  Air  Force  Materiel  Command  (AFMC)  Aeronautical  Systems 
Center  (ASC)  Wright  Laboratory  (WL)  engages  in  technology  transition  and  transfer 
according  to  applicable  public  law,  acts,  executive  orders,  and  Air  Force  policy  directives.9 

1 .3  The  expenditure  of  WL  resources,  in  terms  of  manpower,  facilities,  and  funds,  for  the 
transfer  of  technology  to  companies  and  corporations  has  an  impact  upon  the  mission  of 
the  AFMC  and  the  nation's  economy.  The  measurement  of  the  cost  and  the  impact  to  the 
nation  for  engaging  in  technology  transfer  under  the  referenced  mandated  legislation  has 
been  a  topic  of  numerous  panel  discussions,  university  studies,  industry  concerns,  and  a 
requirement  for  the  Air  Force  Technology  Transfer  Office  (TTO)  Integrated  Product 
Team  (IPT)  as  well  as  the  WL  Plans  Directorate  (WL/XP).  Only  recently  have  affordable 
non-intrusive  cost  metrics10’*11  been  identified  which  theoretically  notes  an  understanding 
of  the  impact  upon  the  nation's  economy  generated  by  technology  transfer  activities  of 
government  (i.e.,  federal)  research  laboratories. 

1.4  Members  of  the  U.S.  Congress,  Departments  of  Defense  (DoD),  Energy  (DoE), 
Transportation  (DoT),  Federal  Laboratory  Consortium  (FLC),  and  industry  partners  are 
the  intended  audience  for  the  results  of  this  PDE  in  Engineering  Management.  The  results 
are  intended  to  assist  all  levels  of  technology  transfer  stakeholders  (e.g..,  managers, 
researchers,  manufacturers)  and  corporate  marketing  functions  with  management  tools 
and  understanding  for  making  product  development  decisions.  The  opportunity  to 
transfer  high  cost  research  and  development  program  technologies  into  new  products 
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solving  many  of  the  difficult  problems  in  the  nation  is  accomplished  through  new  strategies 
focusing  resources  and  benefits  for  mutual  gain  in  food  production,  transportation, 
education,  health  care,  housing,  public  safety  and  national  security.  In  general  the 
technology  sharing  efforts  continue  in  a  cooperative  atmosphere  between  government 
laboratories  and  the  civilian  industry  as  long  as  the  necessary  theories,  concepts,  models, 
tools  and  information  becomes  available  to  measure  the  success  and  anticipate  future  costs 
while  providing  accountability  to  our  nation's  legislature.  By  developing  a  measurement 
prediction  method  based  upon  risk  assessment  techniques,  the  management  of  future 
technology  transfer  programs  will  be  more  effective  and  efficient. 
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2.0  Background 


2. 1  The  need  to  measure  progress  toward  an  objective  arises  as  soon  as  an  objective  is 
established.  If  the  objective  is  at  the  interest  level  warranting  national  attention,  then  the 
determination  of  the  various  attributes  of  a  metric  become  very  important  as  well  as  the 
timeliness  and  intended  use  by  the  customer  seeking  the  information.  ‘Having  established 
the  national  objective  of  increased  technology  transfer  from  federal  laboratories  to  private 
industry,  the  government  is  interested  in  discriminating  between  those  organizations  that 
are  effective  in  technology  transfer  and  those  that  are  not.  The  Executive  and  Legislative 
Branches  of  the  Federal  Government  must  monitor  performance  for  budgetary,  political, 
and  other  reasons.  In  the  case  of  technology  transfer,  private  companies  also  will  monitor 
performance,  if  only  to  determine  whether  they  are  missing  an  opportunity.”12  It  has  been 
proposed  by  Beverly  J.  Berger,  Ph.D.,  the  Washington  D.C.  representative  of  the  FLC  for 
Technology  Transfer  that  creativity  is  required  to  measure  progress  in  technology  transfer. 
Further,  it  will  involve  knowing  whether  cultural  change  is  taking  place  in  the  government 
and  in  the  private  sector  through  objectives  which  provide  both  standards  and  incentives 
for  technology  transfer.  Importantly,  Dr.  Berger  asks  “how  do  we  measure  the  process  of 
technology  transfer  without  distorting  or  impeding  it  ?”13 

2.2  The  March  1995  issue  of  The  Chronicle  of  Higher  Education  presented  findings  to 
the  question  “Does  Technology  Policy  Create  or  Eliminate  Good  Jobs?”14  The 
referenced  study  addressed  measures  of  “...the  economic  benefits  of  a  particular  event  by 
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comparing  the  state  of  the  economy  after  the  event  has  happened  to  the  state  of  the 
economy  assuming  the  event  had  not  happened.”15  The  authors  explore  changes  in  total 
output  produced  for  both  the  long  run  and  short  run  economic  impact  as  well  as  changes 
in  product  mix  influenced  by  market  forces,  self  sufficiency  vs  free  trade  arguments,  and 
issues  of  national  defense  and  economic  security.16  Conclusions  are  reached  about 
technology  transfer  activities  and  their  effect  upon  the  economy.  Although  broad  in 
concept,  the  conclusions  support  a  hierarchy  in  understanding  the  effects  beginning  with 
commercialized  technology,  i.e.,  the  technology  used  by  firms  in  manufacturing,  and 
progressing  through  issues  of  job  creations,  regional  economic  focus  and  national 
economic  focus.17 

2.3  The  scope  of  this  study  focuses  upon  the  Air  Force  WL  technology  transfer  process. 
Beginning  with  a  review  of  the  foundations  of  1980  congressional  legislation,  language 
and  intent  for  “our  country’s  industries,  academia,  and  state  and  local  government 
agencies...”  to  “...  greatly  benefit  from  sharing  our  [federal  laboratories]  technical 
knowledge  and  expertise,”1*  the  study  proceeds  through  the  available  literature  drawing 
upon  a  decade  of  process  attempts  through  legal  instruments,  program  reviews, 
congressional  clarifications ,  legislative  revisions  and  department  directives  establishing  the 
Air  Force  Materiel  Command  (AFMC)  Aeronautical  Systems  Center  (ASC)  Wright 
Laboratory  (WL)  Office  of  Research  Technology  Applications  (ORTA)  and  the  Plans 
Directorate  (XP)  Technology  Transfer  Division  (XPT).  A  specific  look  into  the  processes 
and  activities  implementing  technology  transfer  at  the  functional  level  where  basic 
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research,  engineering  research  and  development,  advanced  demonstration  projects,  and 
transition  to  field  systems  occurrs  results  in  new  insight,  understanding,  tools,  and 
methods  to  the  determine  the  taxpayers’  cost  and  the  nation’s  benefit  from  government 
laboratory  technology  transfer. 

2.4  Initial  findings  from  literature  reviews,  personal  interviews,  and  discussions  note  that 
students  of  technology  transfer  processes  have  defined,  analyzed,  and  proposed  numerous 
findings  and  approaches  for  implementation  instruments,19  factors  affecting  technology 
transfer,20  understanding  the  commercial  sector,21  and  exploratory  understanding  of 
technology  transfer  concepts,  federal  infrastructure,  and  process  models.22  An  extensive 
literature  search,  including  a  personal  interview,  capture  the  essence  of  the  importance  of 
technology  transfer,23  the  status  of  technology  transfer  capabilities  as  established  through 
case  studies,24  and  the  current  status  of  technology  transfer  metrics  development.25  More 
importantly  the  determination  that  management  methodologies  using  these  existing  or 
newly  emerging  metric  tools  for  proactive  technology  transfer  management  were  missing  - 
-  simply  not  addressed  by  the  academic  community.  This  fact  was  not  surprising  since  the 
proposed  development  of  adequate  non-intrusive  metrics  emerged  via  an  academic 
graduate  school  publication26  in  September  1996  and  through  a  personal  interview27 
conducted  during  April  1997.  The  timely  development  of  an  engineering  management 
approach  for  technology  transfer  decision  making  within  WL  has  the  potential  to  define 
and  benchmark  processes  applicable  to  all  government  laboratories. 
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2.5  A  review  of  these  new  technology  transfer  metric  approaches  reveals  the  degree  of 
complexity  and  the  need  for  manageable  non-intrusive  measurement  methods.  The 
Cooperative  Research  Development  Agreement  (CRD A)  information  of  record  within  WL 
has  served  as  a  source  of  data  for  students  from  The  Alfred  P .  Sloan  School  of 
Management  and  the  School  of  Engineering  at  the  Massachusetts  Institute  of 
Technology,2*  and  The  Air  Force  Institute  of  Technology  (AFIT)  Graduate  School  of 
Logistics  and  Acquisition  Management  for  studies  concerning  the  technology  transfer 
process.29  • 301 31* 32  The  work  by  First  Lieutenant  Clinton  J.  Braun  in  1996  investigated  the 
technology  transfer  process  for  opportunities  to  measure  return  on  investment  (ROI) 
based  upon  a  cost/benefit  analysis  model  modified  to  include  intangible  benefits.  His 
review  of  methodologies  for  ranking  projects  included  techniques  for  projected  cash  flow, 
i.e.  payback  periods,  accounting  rate  of  return  (ARR),  net  present  value  (NPV),  and 
internal  rate  of  return  (IRR).33’ 34  The  study  noted  the  difficult  task  of  applying 
conventional  ROI  methodology  to  the  technology  transfer  process.  It  is  highly 
cumbersome  and  intrusive  to  develop  total  estimates  of  a  technology  transfer  project  s 
actual  or  estimated  revenues  and  benefits.  It  is  the  technology  transfer's  intangible  benefits 
among  participating  partners  which  cannot  be  directly  related  to  project  dollars  most  easily 
understandable  by  decision  makers.  The  intangible  benefits  are  often  the  quality  of  life 
improvement  processes  achieved  through  expert  discussion  and  idea  exchanges  which 
assist  an  industry  entity  without  formal  recognition  and  reimbursement.  Intangible 
benefits  are  usually  determined  through  a  heuristic  approach  requiring  significant 
interviewing  with  individuals  in  partnering  organizations.  The  typical  product  of  such 
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intrusions  is  lengthy  qualitative  written  reports  of  the  interview  questions  and  responses. 

In  order  to  overcome  the  problem  that  intangible  benefits  are  sometimes  considered  as 
decision  criteria,  but  many  times  are  overstated,  understated  or  not  quantified,  Lt  Braun 
sought  to  incorporate  a  two  step  technique  of  identifying  and  quantifying  intangible 
benefits  and  assigning  "specified  magnitudes  of  the  value."35  The  identification  and 
quantification  of  intangible  benefits  of  technology  transfer  activities  previous  to  1995  did 
not  exist  in  the  pertinent  literature.  Lt  Braun  investigated  developments  in  the  information 
technology  (IT)  industry  where  the  benefits  for  increasing  productivity  are  intangible  and 
difficult  to  quantify,  i.e.  Oracle  Corporation  development  of  a  model  called  CB-90  which 
assist  managers  with  IT  investment  decisions,  the  Mathias  Schumann  approach  to 
quantifying  intangible  benefits  for  office  automation,36  and  the  methodology  developed  by 
Richard  Pastore  for  information  economics  (EE)  using  a  system  of  quantifying  the 
intangible  benefits  of  a  project  resulting  in  rankings  based  upon  their  expected  contribution 
to  business  objectives.37  Each  of  these  methods  have  limitations  in  their  quantification 
schemes  as  noted  by  Lt  Braun's  thesis.  His  approach  toward  resolution  of  the 
quantification  of  intangible  benefit  schemes  for  technology  transfer  came  from  the 
discipline  of  decision  making  utilizing  probability  encoding  (PE)  interview  processes,  i.e. 
P-method,  V-method,  or  PV-method  where  subjects  respond  to  points  or  values  on  scales 
where  either  probabilities  remain  fixed,  or  values  remain  fixed,  or  in  the  later  case  each 
varies  according  to  the  outcome  or  variable  that  is  being  quantified.38  Lt  Braun  develops  a 
methodology,  which  addresses  both  the  tangible  and  intangible  benefits  of  technology 
transfer,  applicable  for  comparison  of  CRD  A  projects.  Although  a  long  way  from  being  a 


practical,  useful  and  non-intrusive  metric  of  technology  transfer,  the  methodology 
introduced  by  Lt  Braun  does  incorporate  probability  encoding  and  some  of  the  basic  risk 
analysis  techniques  presently  applicable  within  the  state-of-the-art  project  and  performance 
management  See  Appendix  8. 1  Example  CRDA  Benefits  Data.39  Yet  the  end  result  is 
burdensome  to  employ  and  is  limited  to  assessing  technology  transfer  activities  underway 
or  imminent  with  well  defined  CRDA  work  plans. 

2.6  Independent  of  Lt  Braun's  investigations,  Mr.  Roy  W.  Hale,  WL/XPT,  proposed  a 
metric40  for  technology  transfer  processes  based  upon  the  same  available  set  of  CRDA 
case  files.  This  unpublished  study,  initially  proposed  in  1995  and  drafted  in  1996,  was 
defined  and  investigated  against  a  randomly  generated  notional  database  of  three  federal 
laboratories  creating  forty  CRD  As  each  per  year  over  and  estimated  three  year  period. 

The  metric  is  defined  in  macro  factors  as:  [amount  of  dollars  ($)  invested  by  the  outside 
partner]  divided  by  [amount  of  dollars  ($)  invested  by  the  federal  partner].  The  metric  is 
applied  one  time  at  the  mid-point  of  the  agreement  life  and  applicable  to  technology  at 
various  levels  of  maturity  as  demonstrated  by  simulations  generated  using  statistical 
process  control  software  by  SPSS  Inc.41  The  interpretation  results  and  the  conclusions  of 
Mr.  Hale's  proposal  and  investigations  through  his  eventual  application  to  actual  case 
studies  represents  the  simple  non-intrusive,  e.g.  easy  to  obtain  data  from  industry  partner 
and  government  unit,  for  eventual  verification  of  any  future  technology  transfer 
management  prediction  or  decision  making  tools. 


2.7  Not-for-profit  academic  institutions  which  operate  technology  research  centers 
interfacing  with  industry  are  similarly  task  to  determine  their  value  and  worth  to  the 
overall  university  or  college.  The  inquiries42  of  Ashley  Stevens, ‘Boston  University 
Community  Technology  Fund  Director,  into  the  measurement  of  technology  transfer 
activities  related  to  not-for-profit  transactions  provides  the  final  piece  of  background 
information  useful  to  this  PDE.  Stevens  sites  the  difficulty  with  most  technology  transfer 
measures  involving  royalty  income  from  licensing  and  patent  fees  as  reflecting  the  results 
of  transactions  completed  many  years  earlier  and  does  not  present  a  measure  sufficiently 
sensitive  of  year-to-year  changes.  Stevens  states,  "I  believe  that  the  correct  performance 
measure  of  a  technology  transfer  office  is  the  current  total  value  to  the  institution  of  the 
transactions  concluded  in  the  year."43  Stevens  continues  to  propose  that  the  best  method 
to  determine  the  total  value  to  the  institution  is  a  simple  and  ideally  used  method  of 
valuing  a  transaction  as  the  total  pre-commercial  value,  the  sum  of  all  the  pre-commercial 
payments  then  apply  a  risk  adjusted  net  present  value  (raNPV)  method  for  valuing 
technologies.  The  essence  of  this  approach  involves  "constructing  a  timeline  of  the 
development  pathway  that  the  technology  must  follow  to  reach  the  market,  identifying 
some  of  the  key  milestones  and  move  on  to  the  next  stage.  By  overlaying  the  financial 
structure  of  the  transaction  with  this  probability  map,  you  obtain  the  probability  that  you 
will  actually  receive  the  payment ...  obtain  the  net  present  value  by  discounting  back  to 
today  using  a  'cost  of  money*  discount  rate."44  Stevens  concludes  that  these  calculations 
preformed  with  a  simple  Excel  spreadsheet  should  be  executed  before  transaction 
negotiations  and  are  of  value  since  a  basis  then  exist  which  allows  quantification  of  the 
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impact  of  trade-off  opportunities  which  typically  emerge.  Considering  the  institution  as  a 
whole,  the  collective  constructed  time  line/probability  grid  for  all  the  transaction  involving 
technology  transfer  could  be  added  up  as  a  portfolio  for  obtaining  a  final  valuation.  An 
annual  picture  of  the  valuation  would  simply  show  an  increase  or  decrease  of  the  net 
value. 

2.8  The  investigations  of  First  Lieutenant  Braun,  the  theoretical  proposed  metric  and 
simulations  of  Mr.  Hale,  and  the  inquiries  of  Ashley  Stevens  together  benchmark  the 
current  "state-of-the-science-and-art"  of  metrics  for  assessing  the  historical 
accomplishment  of  federal  research  laboratory  projects  or  not-for  profit  institute  programs 
involved  with  technology  transfer.  The  next  challenge  is  to  provide  laboratory 
management  the  decision  rtaking  prediction  methodology  and  tools  applicable  for 
assessing  the  technology  transfer  benefit  available  from  current  or  planned  primary 
mission  related  research.  Once  such  methodology  and  associated  engineering 
management  tools  are  developed  and  administered,  then  the  above  candidate  metrics  have 
truly  value  added  importance  to  research  organizations.  The  goal  of  this  PDE  is  to 
develop  a  management  methodology,  founded  upon  interdisciplinary  human  factors  and 
statistical  processes  which  would  allow  metrics  to  become  useful  management  tools. 


27 


3.0  Methods 


3.1  Methodology 

3.1.1  The  research  plan  for  this  study  used  action  oriented  experiential  research  and 
learning,45  probability  encoding4*  combined  with  the  experimentation47  statistical  methods4* 
championed  by  Dr.  Donald  J.  Wheeler  and  used  extensively  in  the  Adaptation  of  the  Air 
Force  Materiel  Command  fAFMO  Metrics  Course49,  and  modeling  and  simulation.  The 
method  employed  in  the  research  leading  to  the  PDE  recognizes  that  technology  transfer 
occurs  through  a  management  system  possessing  internal  processes  which  take  inputs  and 
generatesoutputs  under  a  set  of  constraints  imposed  from  various  sources.  As  do  many 
systems  which  exhibit  natural  variations,  the  process  of  technology  transfer  is  no  different. 
Natural  variations  in  input  lead  to  variations  in  output.  A  diverse  population  of  engineers 
and  scientists  evelop  technology  for  specific  roles  and  missions.  The  technology  transfer 
opportunity  provides  for  the  diversion  of  military  technology  into  new  civil  market 
products  whose  acceptance  and  product  use  is  also  a  function  of  the  interest,  desires,  and 
whims  of  a  diverse  population  of  investors,  buyers,  and  users.  The  influence  of  the  input 
on  the  output  is  not  well  understood  and  is  subject  to  uncertainty.  Testing  educated 
estimates  based  upon  known  facts  is  the  only  opportunity  to  generate  meaningful 
information  for  understanding  the  system  or  process  by  which  technology  is  transferred; 
however,  it  requires  knowledge  of  the  imposed  constraints  and  their  relationship  to  an 
established  strategy.  Risk  analysis  represents  a  field  of  study  which  when  combined  with 
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encoded  predictions  of  expert  judges  offers  development  of  a  cost  estimate  for  technology 
transfer  efforts  to  manage  decision  making  processes.  It  is  this  predicted  technology 
transfer  cost  estimate  we  seek  for  a  selected  focused  group  of  technologies  developed  by 
WL  to  provide  a  management  tool  guiding  future  resource  expenditures. 

3.1.2  The  method  of  research  for  this  PDE  is  grounded  in  the  residency  seminars,  new 
learning  and  internship  which  resulted  in  a  strategy  plan  development  experience50 
generating  an  understanding  of  the  technology  transfer  process.  The  attainment  of  the 
goal  stated  in  Section  2.8  would  result  in  the  development  of  a  new  management 
prediction  tool  verifiable  through  the  selection  of  an  appropriate  metric.  It  is  important  to 
note  that  the  definition  of  the  word  "metric"  contained  in  the  glossary  of  this  document 
includes  the  key  wording  "drive  appropriate  action  and  must  be  linked  to  a  strategic  plan." 
Through  fulfillment  of  The  Union  Institute  internship  residency  requirement  for  The 
Doctor  of  Philosophy  in  Engineering  Management,  a  review  of  current  technology  transfer 
processes  was  completed  and  a  new  more  efficient  and  effective  strategy51  with 
appropriate  task  plan  (Appendix  9.2)  for  implementation  of  an  out  reach  program  was 
developed  and  executed.  The  process  addressed  concerns  of  congressional  mandates  and 
federal  law52  affecting  WL.  It  generated  a  new  approach  for  technology  transfer  through 
out  reach  programs53.  The  creation  of  this  new  focus  generated  the  authority  and 
direction  to  organized  the  WL  CRDA  data  files  into  statistically  useful  information 
(Appendix  8.3)  for  validating  selected  metrics.  Ultimately  this  data  set  is  of  interest  for 
consideration  by  any  theorist  attempting  to  produce  engineering  management  tools 
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capable  of  assessing  current  technology  programs  not  yet  experiencing  technology 
transfer  efforts.  The  theoretical  approach  chosen  introduces  risk  management  analysis 
tools54, 55, 561 S7, 58, 59,  ^ 61, 62, 63,  65, 661 67, 68  to  produce  the  desired  performance  indicators  for 

technology  transfer  efforts  tied  to  a  strategy  for  technology  transfer.69 

3.2  Theoretical  Approach 

3.2. 1  The  theoretical  approach  in  this  study  draws  from  the  quality  process  design  of 
experiments  scientific  method  for  investigation  of  multifactored  inputs  affecting  the  output 
of  a  particular  process.  Within  the  design  of  experiments  approach  to  the  scientific  method 
recognized  that  the  output  of  any  process  is  affected  by  all  possible  factor  combinations  of 
inputs.  Infact  a  factorial  relationship  usually  exist  generating  unmanageable  testing  in 
many  situations  often  resorting  in  the  development  of  a  random  test  matrix.  Even  the 
process  of  considering  one  factor  at  a  time  while  holding  all  other  factors  constant 
generates  a  large  expensive  testing  and  analysis  requirement  upon  the  research 
investigator.  The  design  of  experiments  (DOE)  approach  based  upon  the  work  of  George 
Box,  Charles  Edward  Deming,  and  Genichi  Taguchi70  provides  the  third  option  for 
analyzing  cause  and  effect  which  incorporates  knowledge  and  judgement,  group  problem 
solving  techniques,  and  statistical  tools.  It  requires  adherence  to  the  scientific  method, 
viewed  as  a  process.  The  general  construct  or  model  views  the  output  of  an  experiment  as 
being  affected  by  multiple  inputs,  in  which  the  influence  of  the  inputs  to  the  output  is  not 
well  understood.  During  the  design  of  the  experiment  the  pretest  review  addresses  the 
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process  to  be  studied,  identification  of  response  variable(s)  that  the  customer  observes,  a 
clear  statement  of  the  objectives  in  terms  of  the  response  variable(s),  identification  of  the 
inputs  and  a  selection  of  the  interaction  issue  of  the  inputs.  The  use  of  DOE  statistical 
tools  requires  that  the  response  variables  be  measurable  with  numbers  on  a  continuous 
scale  and  that  we  must  be  able  to  distinguish,  separate  and  control  the  factors,  e.g.  inputs, 
which  have  independence.  For  this  approach  to  be  used  in  generating  a  term  useful  in  the 
prediction  of  potential  future  technology  transfer  CRD  As  from  current  R&D  programs 
requires  special  development  of  a  questionnaire  and  expert  judgement  response.  The  use 
of  expert  judges  to  evaluate  the  WL  R&D  programs  in  terms  of  known  metrics  while 
providing  an  estimated  guess  as  to  the  potential  yes  (+1)  or  no  (-1)  for  future  successful 
technology  transfer  possibilities  as  an  outcome  of  a  natural  processes  exits  through  PE 
research  interview  processes. 

■r 

3.2. 1  PE  as  a  reserach  interview  process  allows  for  the  opportunity  to  extract  and  quantify 
individual  judgements  about  outcomes.  PE  has  grown  in  popularity  and  is  now  commonly 
used  in  probabilistic  decision  analysis  where  uncertainity  can  be  incorporated  through  the 
assignment  of  probability  distributions  to  the  variables  or  outcomes.  By  using  PE  the  risk 
involved  in  quantifying  responses  from  a  test  subject  or  expert  judge  can  be  accounted. 
Expert  judgement  or  opinion  is  incorporated  into  the  process,  numbers  are  generated,  and 
a  probability  distribution  can  be  constructed  for  the  provided  estimates.  The  guidelines  for 
using  PE  in  the  design  of  the  experiment  are:  i)  choose  only  uncertain  quantities  that  are 
important  to  decision,  ii)  define  the  quantity  as  an  unambiguous  state  variable,  iii) 
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structure  the  quantity  carefully  so  that  it  is  understandable  to  the  expert  subject,  iv) 
clearly  define  the  quantity;  and  v)  describe  the  quantity  using  a  scale  that  is  meaningful." 
From  the  data  generated  using  these  guidelines  for  questionaire  development "... 
reasonable  estimates  and  distributions  can  be  constructed  from  uncertain  quantitities."71 
PE  minimizes  the  effect  of  human  expert  judgment  which  exhibits  individual  bias,  either 
displacement  bias,  e.g.  shift  in  distribution  vertically,  or  variability  bias,  e.g.  change  in 
shape  of  distribution.  This  has  been  demonstrated  and  documented  in  literature  via  the  PE 
method  question  development  process  based  upon  expert  learned  knowledge  criteria.72 

3.2.2  The  specific  achievable  goal  in  the  development  of  the  engineering  management 
tool  for  technology  transfer,  is  the  development  of  a  method  to  estimate  the  WL 
technology  transfer  investment  anticipated  for  facilitating  the  movement  of  technology 
from  current  R&D  programs  into  future  commercial  products.  Based  upon  previously 
proposed  metrics  presenting  historical  technology  tansfer  process  data  for  WL  programs, 
expert  judges  experientially  identify  active  R&D  programs  according  to  the  developed 
strategy  for  generating  focused  out  reach  groups.  Risk  assessment  is  introduced  through  a 
questionaire  response  incorporating  a  modified  Air  Force  acquisition  technology  risk 
model.  The  model  introduces  WL  technology  transfer  investment  cost  based  upon  past 
histroical  data  as  experienced  by  the  Air  Force  unit  while  incorporating  schedule  variations 
and  uncertainity  forcast  interpreted  as  a  risk  factor.  It  is  important  to  note  that  the  use  of 
the  expert  judge  panel  involves  the  PE  neccessary  components  of  motivation,  structuring, 
conditioning,  encoding,  and  verification  in  the  initial  process  identifing  potential  WL  R&D 
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programs.  This  process  was  key  to  data  collection  during  the  assessment  phase  of  this 
research. 

3.2.3  The  classical  motivation  phase  introduces  the  potential  expert  judge  to  the 
background  and  purpose  of  the  decision  making  study.  The  motivation  phase  of  this 
experimentwas  accomplished  through  the  selection  of  technology  transfer  focal  point 
individuals  within  each  directorate  of  WL  who  participated  in  the  technology  transfer 
strategy  plan  development  and  who,  in  addition,  had  experience  in  the  development  of 
CRD  As  between  Air  Force  units  and  industry  partners. 

3.2.4  The  structuring  phase  of  the  PE  process  occurred  during  the  creation  of  a 
questionnaire  based  upon  aviable  data  from  performance  tracking  measures  of  the  CRD  As 
as  well  as  the  definition  of  the  WL  investment  in  the  technology  transfer  activity  as  a 
numerical  sum  of  scientist  and  engineer  (S&E)  encumberd  salaries  converted  in  FY97 
dollars  plus  any  charges  for  utilization  of  WL  facilities  as  contained  in  the  WL  program 
management  tracking  database  called  "A  Science  &  Technology  Action  Reporting  System 
(ASTARS)."  ASTARS  contains  2,061  active  R&D  job  order  number  data  files. 

3.2.5  The  conditioning  phase  of  the  PE  process  occurred  through  the  selection  of 
currently  funded  and  active  R&D  programs  contained  in  ASTARS  which  most  nearly 
matches  the  technology  transfer  strategy  plan.  The  WL  technology  transfer  focal  points, 
i.e.  strategy  plan  team  members,  which  were  selected  as  expert  subjects  for  responding  to 
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the  questionnaire,  generated  the  listing  of  current  technology  programs  culled  from  the 
WL  database  applicable  to  the  technology  transfer  out  reach  to  general  aviation  (ORGA) 
focus  group.  Specifically,  the  experts  utilized  the  National  Aeronautics  and  Space 
Administration-Industry-Federal  Aviation  Administration  Advanced  General  Aviation 
Transport  Experiments  (AGATE)  Operation  Requirements  Document.73  The  experts 
were  guided  by  their  knowledge  of  the  strategy  task  plan  into  thinking  fundamentally 
about  judgment  and  bias  avoidance. 

3.2.6  The  encoding  phase  of  the  study  quantified  the  important  judgment  of  the  expert 
choices  in  probabilistic  terms  which  the  verification  review  cross  checked  the  validation  of 
the  selected  research  and  development  programs.  The  estimated  potential  dollar  amounts 
of  the  WL  investment  into  the  technology  transfer  process  for  each  program  were  summed 
for  all  expert  judges  and  represent  a  collective  outcome  response  in  the  DOE  approach. 
An  overall  estimate  of  the  WL  technology  transfer  investment  for  the  R&D  programs  were 
subsequently  summed  for  a  total  Air  Force  investment  value  from  the  technology  transfer 
ORGA  suite  of  programs.  When  considered  as  an  anticipated  average  per  individual 
CRD  A,  the  dollar  amount  was  compared  to  proposed  metrics  presenting  the  past 
historical  data. 

3.2.7  The  development  of  an  appropriate  technology  transfer  risk  assessment  schedule 
for  impact  upon  cost  was  necessary.  The  risk  factor  values  chosen  by  this  Learner  for 
technology  transfer  are  found  in  the  Air  Force  Quantified  Risk  Analysis  (QRA)  Criteria  of 
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1993.  This  document  concluded  that  risk  categories  for  acquisition  management 
technical/performance  risk  analysis  include  requirements,  technology,  engineering, 
manufacturing,  support  and  managment.74  The  categories  of  technology,  engineering,  and 
manufacturing  are  technical  in  nature  and  address  expert  assessment  in  terms  of  questions: 
i)  Is  the  technology  [under  consideration]  available  and  proven  in  previous  use?  ii)  How 
much  new  design  is  needed  to  achieve  requirements?  iii)  Are  the  required  manufacturing 
processes,  facilities,  and  sources  of  materials  known  and  available?  Additionally,  the 
Defense  Systems  Management  College  (DSMC)  expands  the  description  of  technical  risk 
to  include  issues  of  complexity/difficulty  in  meeting  requirements,  percent  proven 
technology,  experience  in  the  field  needed,  lack  of  work  on  similar  programs,  special 
resources  needed,  operating  environment,  required  theoretical  analysis,  degree  of 
difference  from  existing  technology ,  numerous  state-of-the-art  advances,  lack  of 
supporting  state-of-the-art  advances,  integration/interface,  reliability,  and  maintainability.75 
The  DSMC  guide  depicts  the  important  relationship  between  technical,  cost,  and  schedule 
risk  in  the  following  figure76. 


Figure  1 .  Risk  Facets 
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3.2.8  Measuring  cost  risk,  schedule  risk,  and  technical  risk  presented  unique  challenges. 
Cost  and  schedule  in  terms  of  dollars  and  time  are  often  combined  using  established 
statistical  rules  and  techniques.  Technical  risk  is  diverse  and  no  common  unit  of  measure 
is  fully  accepted  by  industry  and  government.  The  combination  of  technical  risk  in  weight, 
speed,  range,  weight,  capacity,  complexity,  reliability,  safety,  temperature,  vibration,  etc. 
are  specific  to  technologies  and  the  application.  However,  students  of  risk  assessment 
have  developed  methods  to  assess  and  describe  technical  risk  through  various  specialized 
approaches:  Narrative,  Risk  Templates  ("Willoughby  Templates"),  Qualitative,  Risk 
Scoring,  Risk  Scales,  Maxwell  Risk  Criteria  Manx,  Technical  Performance  Measures,  and 
System  Maturity  Matrix.77  The  QRA  risk  factors  for  technology  in  engineering  acquisition 
management  programs  have  traditionally  been  defined  as: 

0.01  Operational  and  deployed 

0.20  In  use  by  another  program 

0.50  Technology  transition  experiments  successfully  completed 

0.70  Initial  proof-of-concept  experiments  successfully  completed 

0.90  Basic  research  only.  No  development  work 
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The  risk  definitions  required  modification  (See  Appendix  9.5)  for  this  PDE.  This 
modification  was  accomplished  prior  to  the  probability  encoding  interview  questionnaire 
distribution  to  the  expert  judges.  The  modification  was  necessary  due  to  the  unique  nature 
of  technology  transfer  not  being  the  primary  mission  goal  of  the  individual  R&D  program 
under  assessment.  The  questionnaire  for  this  study  was  developed  using  cost  information 
provided  through  the  metric  development  process  previously  discussed  and  the  QRA  Risk 
Factors  along  with  the  modified  definition  for  technology  transfer  by  this  Learner.  Each 
expert  required  a  prebrief  about  the  assessment  and  the  process  by  which  they  qualified  as 
an  expert,  i.e.,  experience  with  WL  R&D,  CRDA,  the  WL  Technology  Transfer  Strategy 
Plan,  the  "balloting"  of  existing  CRDAs  determining  out  reach  program  affiliation,  and 
general  familiarity  with  WL/XPT  efforts  to  develop  suitable  metric(s)  evaluating  the 
success  of  technology  transfer  activities.  The  qualification  of  the  expert  judges  was 
accomplished  within  the  motivation,  structuring,  and  conditioning  requirements  for  PE 
prior  to  the  collection  of  data.  Each  expert  judge  was  expected  to  complete  the 
questionnaire  addressing  individually  the  selected  R&  D  programs  from  within  their 
respective  WL  Directorates  (i.e..  Avionics,  Flight  Dynmanics,  Manufacturing  Technology, 
Materials,  and  Propulsion  &  Power). 

3.2.9  The  responses  of  the  expert  judges  collectively  represents  a  notable  data  set  from 
which  significant  information  was  determined  utilizing  a  DOE  approach78  to 
understanding  the  relationship  of  cost,  schedule,  and  risk  for  technology  transfer.  The 
composite  means  for  cost  as  affected  by  forcasted  schedule  and  risk  judgements 
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represented  the  "treatments"79  and  the  result  of  completing  the  DOE  process  would 
resulted  in  a  linear  algebraic  relationship  predicting  the  WL  invesment  funding  required  to 
accomplish  technology  transfer  CRD  As. 

3.2. 10  Applying  this  method  to  a  specific  set  of  cost,  schedule,  and  time  data  from 

active  R&D  programs  applicable  to  the  Outreach  to  General  Aviation  strategy  produced 

an  estimate  for  future  WL  investment  in  technology  transfer.  Subsequent  application  to  all 

focused  out  reach  programs  in  technology  transfer  would  represent  a  composite 

management  prediction  for  risk  adjusted  investment  to  accomplish  the  CRDA  process. 

The  estimated  investment  term  eventually  could  be  compared  to  the  median  denominator 

term  from  the  metric  described  in  paragraph  2.6  Including  uncertainty  in  a  given  project 

through  risk  management  produces  potential  for  forming  the  management  bridge  from 

* 

understanding  historical  past  events  (metrics)  to  the  future  prediction  of  possible 
outcomes  grounded  in  strategic  planning  and  uncentainity  expectations. 

3.2. 11  A  review  of  Wright  Laboratry  CRDA  files  was  accomplished  by  ten  (10)  WL 
technology  transfer  specialist  and  focal  point  representatives  through  a  review  and 
balloting  process  (Appendix  9.3).  Those  agreements  which  clearly  represented 
technologies  aligned  with  the  task  plan  for  the  ORGA  were  identified  and  listed  by 
industry  partner  name  and  CRDA  number.  Significantly,  a  data  set  relative  to  a  strategy 
plan,  now  existed  for  use  in  developing  new  decision  making  and  management  tools. 

Seven  (7)  of  the  ten  (10)  WL  technology  transfer  focal  point  scientist  and  engineers  who 
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participated  in  the  development  of  the  WL  Technology  Transition  and  Transfer  Strategic 
Plan80  and  the  selection  of  focused  technologies  for  the  ORGA  were  provided  an  initial 
review  of  historical  CRD  A  metric  results81  presented  by  Mr.  Bill  Hale.  Based  upon  their 
strategic  planning  experience,  familiarity  with  the  task  planning  for  the  focused  ORGA, 
and  the  CRD  A  metric;  the  expert  judges  were  presented  with  a  selection  of  currently 
active  technology  research  and  development  (R&D)  programs  for  review  and  execution  of 
the  questionnaire.  This  review  was  accomplished  in  accordance  to  the  methods  of  PE 
which  attempted  to  match  the  strategic  plan  goals  with  the  expert's  opinion  as  to  potential 
for  future  CRDA  development. 

3.2.12  This  review  selecting  technologies  appropriate  to  the  revitalization  efforts  of  the 
U.S.  general  aviation  industry  involved  the  WL  ASTARS  database  containing  2,061 
unclassified  R&D  programs.  The  database  was  searched  and  a  report  generated  which 
identified  a  subset  of  135  current  or  recently  completed  R&D  technology  programs. 

These  programs  met  future  Air  Force  mission  requirements  while  also  matching  the 
technology  strategy  task  plan  goals  for  the  Technology  Tranfer  ORGA.  Finally  only  open 
and  active  R&D  programs  were  retained  in  the  study  with  recent  completed  efforts 
eliminated  from  future  consideration.  The  remaining  fifty  (50)  R&D  programs  which  were 
identified  by  Job  Order  Number  (JON),  schedule,  funding,  and  technology  transfer 
potential  ratings  are  contained  in  Appendix  9.4.  These  programs  prepresent  the  WL 
ORGA  focused  technology  transfer  opportunities.  The  study  to  determine  the  applicable 
information  of  significance  to  the  strategy  plan  was  completed  and  prepared  for  a  Taguchi 
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Design  of  Experiment  (DOE)82  testing  methodology.  The  Taguch  DOE  method  involves 
processing  experimental  data,  in  this  case  the  expert  judge  panel  PE  questionnaire 
response  data,  though  an  efficient  set  of  DOE  component  tables,  charts  and  arrays,  i.e., 
data  averages  table,  factors  pivot  table,  pretest  potential  interactions  array,  L8  orthogonal 
array,  L8  column  assignment  and  results  chart,  linear  model  calculating  coefficient  results 
table,  and  finally  the  resulting  linear  model  result  equation  chart.83  The  theory  of  the  DOE 
method  is  well  documented  in  literature  and  used  extensively  throughout  the  world  in 
private,  university,  and  government  laboratory  experimental  research  projects. 

3.2.13  A  Probability  Encoding  (PE)  interview  questionnaire  was  prepared  (See  Appendix 
9.5)  and  distributed  along  with  ASTARs  data  including  R&D  program  objective, 
approach,  and  status  descriptions  to  the  expert  judges  on  3  -  4  June  1997.  Individal 
interview  pretest  discussion  with  each  expert  judge  was  accomplished  by  the  Learner. 

The  responses  from  the  expert  judges  were  returned  on  or  before  12  June  1997. 


40 


4.0  Findings 


4. 1  The  encoded  results  are  presented  in  Appendix  9.6.  The  data  in  column  three  (3) 
prepresents  estimated  possibility  that  the  R&D  program  would  (Yes  =  +1)  or  would  not 
(No  =  -1)  produce  revenue  generating  technology  transfer  activity.  This  column  is  titled 
Technology  Transfer  Possibility  (T2  Psblty).  Note  that  expert  judge  probability  encoding 
pretest  interview  process  determined  that  a  negative  answer  would  still  represent  the 
remote  possibility  that  the  lowest  level  of  technology  transfer  activity  involving  expert 
engineer  consultation  and  advice  from  the  WL  could  potentially  result  in  a  revenue 
generating  CRDA  Previous  experience  with  the  development  of  a  Mechanically 
Optimized  Special  Socket  Wrench  by  The  Main  Corporation  (TMC)  under  CRDA  94- 
272-WL-01with  WL  Flight  Dynamics  Structures  Division  (WL/FBAD)  resulted  in  a  very 
small  amount  of  WL  manpower  resouce  expenditure  yet  produced  revenues  often  cents 
($.10)  for  the  first  100k  sold,  ($.05)  for  the  next  400K  sold,  and  ($.02)  thereafter.  It  is 
estimated  that  sales  could  exceed  several  million  units  per  year.  Thus  negative  responses 
(No)  to  question  one  (1)  on  the  questionnaire  were  encoded  upon  the  data  sheet  as  (-1) 
response  and  accordingly  question  two  (2)  the  estimated  technology  transfer  investment  as 
a  percentage  of  the  actual  R&D  program  cost,  question  three  (3)  the  technology  transfer 
process  schedule  in  months,  and  question  (4)  the  technology  transfer  risk  assessment 
factor  were  encoded  on  the  form  as  A.  0.05%  or  .0005,  A.  Six  (6)  Months,  and  A.  0.01 
Risk  respectively.  The  data  for  question  two  (2)  is  found  in  column  four  (4)  titled 
Technology  Transfer  Investment  Percentage  (T2  Invstmnt  %),  question  three  (3) 
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responses  are  in  column  five  (5)  titled  Technology  Transfer  Months  (T2  Mnths),  and 

question  four  (4)  responses  are  in  column  six  (6)  titledTechnology  Transfer  Risk  (T2 

Risk).  Column  seven  (7)  is  the  calculated  estimate  of  required  WL  technology  transfer 

resources  to  accomplish  a  revenue  generating  CRD  A.  Forty-nine  (49)  of  the  fify  (50) 

questionnaires  were  returned  by  the  expert  judges.  Note:  The  questionnaire  for  JON 

24180260  was  not  returned  while  awaiting  consultation  of  the  expert  judge  with  the 

project  engineer  to  make  the  final  selection  of  the  answers  to  the  questions.  Repeated 

attempts  to  retrieve  the  information  were  unsuccessful  to  secure  the  form  in  a  timely 

period  for  analysis.  The  remaining  49  questionnaires  constitute  a  significant  statistical 

sample  from  the  original  database  of  2,061  R&D  programs  available  for  consideration. 

The  column  seven  (7)  computed  amount  of  technology  transfer  investment  in  thousands 

($K)  is  equal  to  the  original  R&D  program  cost  which  generated  the  technology  for 

* 

military  purposes  times  the  technology  transfer  investment  percentage  plus  the  same 
resulting  amount  times  the  risk  factor: 

Technology  Transfer  Estimated  ($K)  =  R&D  Prgm  ($K)  x  T2  Invstmnt  (%)  x  100  + 
[R&D  Prgm  ($K)  x  T2  Invstmnt  (%)  x  100]  x  T2  Risk  (Nondimensional  Unit) 

This  column  is  labled  Technology  Transfer  Estimated  $  K  (T2  Estmtd  $K). 

4.4  The  Taguchi  Design  of  Experiments84  methods  of  data  processing  involves  the 
following  components  developed  from  the  expert  judge  response  information: 
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Number  of  Yes 

9 

Avg  % 

0.23% 

Avg  Mnths 

10.875 

Avg  Risk 

0.1333333 

Table!.  Data  Averages 
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The  averages  become  the  pivot  points  in  the  Design  of  Experiment  factors  array: 


FACTORS 

Level 

Pivot 

Level 

< 

< 

Possibility 

-1 

9 

1 

T2  $% 

-1 

0.23% 

1 

T2  Mths 

-1 

10.875 

1 

T2Risk 

-1 

0.1333333 

1 

Table  2.  Factors  Pivot  Table 


The  Design  of  Experiments  pretest  determination  of  the  potential  interactions  are 
anticipated  and  assigned  to  the  following  matrix: 


PRETEST 

POTENTIAL 

INTERACTIONS 

T2  Psblty 

T2  Invstmnt  % 

T2  Mnths 

T2  Risk 

T2  Psblty 

YES 

YES 

NO 

T2  Invstmnt  % 

lllillllllllllllllllllllllllilllll 

YES 

YES 

T2  Mnths 

IIIIIIIIIIIIIIIIIIIIIIIIIIIIIIIIII 

YES 

Table  3.  Pretest  Potential  Interactions 
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The  Taguchi  Design  of  Experiment  L8  orthogonal  array  assignment  for  the  responses 
based  upon  the  prestest  potential  interactions  follows: 


L8 

C2 

C3 

C4 

C5 

C6 

C7 

Cl 

3 

2 

5 

4 

7 

6 

C2 

1 

6 

7 

4 

5 

C3 

7 

6 

5 

4 

C4 

1 

2 

3 

C5 

3 

2 

C6 

1 

Table  4  Taguchi  Design  of  Experiments  L8  Orthogonal  Array 


L8 

GM 

P 

% 

P% 

1 

R% 

RP 

R 

EJ  Grp  #1 

EJ  Grp  #2 

EJ  Grp  #N 

#1 

1 

-1 

-1 

-1 

-1 

-1 

-1 

-1 

36.92358 

#2 

1 

-1 

-1 

-1 

1 

1 

1 

1 

0 

#3 

1 

-1 

1 

1 

1 

1 

-1 

-1 

0 

#4 

1 

-1 

1 

1 

-1 

-1 

1 

1 

0 

#5 

1 

1 

1 

-1 

-1 

1 

1 

-1 

0 

#6 

1 

1 

1 

-1 

1 

-1 

-1 

1 

111.2725 

#7 

1 

1 

-1 

1 

1 

-1 

1 

-1 

0 

#8 

1 

1 

-1 

1 

-1 

1 

-1 

1 

0 

Table  5  L8  Column  Assignments  and  Results  of  Expert  Judge  Group  #1 
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The  linear  model  calculating  coefficients  are  determined  from  the  L8  column  assignments  : 


Cl  = 

18.52451 

lllllllllll!!llllllll!lll!ll!llll!lllllllll 

iiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiimii 

C2  = 

9.293615 

p 

-1 

C3  = 

9.293615 

% 

-1 

C4  = 

-18.52451 

p% 

-1 

C5  = 

9.293615 

T 

1 

C6  = 

-18.52451 

R% 

-1 

C7  = 

-18.52451 

RP 

-1 

C8  = 

9.293615 

R 

-1 

Table  6  Linear  Model  Calculating  Coefficients 

** 

The  linear  model  produces  the  equation  which  results  in  the  determination  of  the  estimated 
WL  investment  for  technology  transfer  necessary  to  achieve  an  anticipated  nine  (9) 

CRD  As  for  the  ORGA  from  current  R&D  programs: 


T2  ($K)  =  C1+C2*P+C3*%+C4*P%+C5*T+C6*R%+C7*RP+C8*R  = 

55.51081 

Average  T2  Estimated  Investment  per  Anticipated  Active  (Yes  +1)  CRD  As  = 

6.167867778 

Table  7  Linear  Model  Result:  Wright  Laboratory  Technology  Tranfer  Estimated 
Investment  for  the  Out  Reach  to  General  Aviation 
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5.0  Interpretation  &  Limitations 


5.1  The  calculated  average  technology  transfer  estimated  risk  adjusted  resource 
investment  per  nine  (9)  CRD  As  anticipated  to  occurr  from  the  set  of  49  selected  R&D 
programs  is  determined  to  be  $  6, 1 67.87.  The  aggrate  total  anticipated  risk  adjusted 
resources  for  the  ORGA  focused  technology  tranfer  effort  is  $  55,510.81  in  fiscal  year 
(FY)  1997  dollars.  These  results  represent  a  single  point  in  time  evaluation  obtained  in  the 
least  most  intrusive  manner  to  the  WL  Scientists  and  Engineers. 

5.2  The  resulting  linear  equation  model  for  predicting  the  risk  adjusted  aggregate  total 
for  the  strategy  plan  out  reach  program  prepresents  the  capability  of  the  Taugchi  Design 
of  Experiments  Method  to  analized  data  from  minimally  populated  responses  to  achieve 
significant  results.  The  combination  of  the  probability  encoding  questionnaire  technique 
for  obtaining  the  responses  of  qualified  expert  judges  still  often  result  in  a  set  of  sparse 
data.  Table  5  column  ten  (10)  which  is  identified  as  Expert  Judge  Group  #1  (EJ  Grp  #1) 
resulted  in  only  two  entries  calculated  from  appropriate  column  data  (Appendix  9.6)  based 
upon  the  Design  of  Experiments  L8  Orthorgonal  Array. 

5.3  The  accuracy  of  the  estimated  risk  adjusted  technology  transfer  resource  investment 
amount  can  be  tracked  by  the  simple  metric  of  Job  Order  Number  (JON)  cost  charged  to 
CRD  As  which  develop  in  the  future  from  the  identified  R&D  programs  in  this  study.  To 
obtain  a  near  term  understanding  for  the  success  achieved  by  the  linear  model  developed 
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through  the  results  of  the  Design  of  Experiments  approach  we  look  to  the  previously 
mention  work  of  Mr.  Roy  W.  Hale  (  Paragraph  2.6).  Mr.  Hale's  proposed  Technology 
Transfer  Metric  Leverage  Factor  is  defined  as:  [amount  of  dollars  ($)  invested  by  the 
outside  partner]  divided  by  [amount  of  dollars  ($)  invested  by  the  federal  partner].  In  the 
case  of  WL,  the  denominator  of  this  leverage  factor  would  be  an  actual  calculated  amount 
of  the  from  CRDA  JON  cost  acount  database.  It  is  important  to  recall  that  the  expert 
judges,  who  participated  in  this  study,  estimating  the  future  risk  adjusted  WL  technology 
transfer  investment  were  the  same  experts  who  reviewed  over  100  current  files  during  the 
selection  of  CRDA's  catagorized  into  the  strategy  plan  focused  out  reach  efforts,  i.e. 
automotive,  general  aviation,  medical,  public  safety,  agriculture,  etc.  This  CRDA 
catagory  selection  process  was  an  integral  part  of  the  PE  reqired  motivation,  structuring, 
conditioning,  and  verification  research  interview  components.  The  selection  of  the  current 
CRDA's  applicable  to  the  ORGA  are  found  in  Appendix  9.4  This  set  of  historical  data 
represents  an  approximation  to  information  available  for  comparison  with  the  newly 
developed  risk  adjusted  WL  technology  tranfer  investment  estimate  for  future  R&D 
programs.  Twenty-four  (24)  of  sixty-one  (61)  CRDA's  had  available  mid-term  data  for  use 
in  calculating  the  metric  technology  transfer  leverage  factor.  This  single  point  assessment 
as  of  May  2, 1997  for  the  applicable  ORGA  CRDA's  determined  that  the  median  WL 
investment  per  CRDA  was  $  6,045.00  while  the  mean  was  $  10,585.50.  Comparing  these 
investment  terms  from  current  general  aviation  related  WL  CRDA's  to  the  estimated  mean 
for  future  potential  general  aviation  CRDA's  from  the  WL  R&D  programs  ( $  6,167.87), 
provides  confidence  that  future  tracking  of  WL  CRDA  cost  data  against  the  estimated 
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technology  tranfer  investment  is  warranted. 


5.4  There  are  limitations  to  this  PDE  developed  management  methodology  resulting 
from  the  combination  of  a  metric,  strategic  plan,  PE,  risk  management,  and  the  Taguchi 
DOE  technique  for  program  management.  Although  the  metric  may  be  as  simple  or 
complex  as  the  user  desires  it  must  be  clearly  related  to  the  strategic  plan  of  the 
organization  seeking  to  implement  the  management  methodology.  A  strategic  plan 
development  or  review,  study,  and  evaluation  must  involve  the  cadre  of  potential  expert 
judges  prior  to  the  PE  interviewing  research  data  collection  process.  The  candidate  expert 
judge  involvement  within  the  implementing  organization’s  strategic  planning  processe 
fulfills  the  PE  expert  judage  pannel  qualification  requirements.  Additionally,  the  interview 
questionnaire  must  be  simple  and  non-intrusive  while  obtaining  data  definable  within  the 
DOE  interaction  array  of  possibilities.  Finally,  a  limitation  of  the  DOE  technique  is  found 
in  its  application  to  cause  and  effect  relationships  of  experimental  processes. 

Additionally,  in  this  PDE  one  set  of  data  was  not  returned  in  a  timely  manor  to  be  included 
in  the  analysis.  Subsequent  last  minute  inspection  of  the  missing  data  noted  that  an 
additional  cell  in  Table  5  would  have  been  generated  increasing  the  final  calculated  value 
reported  in  Table  7  achieving  an  improved  correlation  with  the  leverage  factor 
denominator  of  the  chosen  technology  transfer  metric. 
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6.0  Conclusions  &  Recommendations 


6. 1  The  management  of  technology  transfer  process  now  has  available  a  new 
methodology  for  key  decisions  makers.  The  effective  metric85, 86  which  measures  the  cost 
of  the  conversion  process  from  research  to  manufacturing  now  combined  with  a  prediction 
method  incorporating  risk  assessment  of  new  technology  for  future  transfer  agreements 
resulting  from  this  PDE.  These  affordable  non-intrusive  methods  which  rely  heavily  upon 
the  science  of  statistics  within  the  PE  and  the  Taguchi  DOE  methods  of  research  provide 
new  tools  necessary  for  understanding  cost  and  impact  of  technology  transfer  activities  of 
federal  research  laboratories. 

6.2  The  opportunity  to  involve  additional  expert  judge  panels  composed  of 
representatives  from  each  directorate  would  enhance  the  quantity  of  data  by  providing 
additional  calculated  entries  into  the  columns  reserved  for  EJ  Grp  #  2  through  EJ  Grp  #  N 
of  Table  5.  However;  increasing  the  number  of  expert  judge  panels  increases  the  intrusion 
into  the  work  force  routine  potentially  introducing  an  artifical  disruption  of  the  process 
being  investigated.  Future  meaningful  studies  appropriate  for  a  Doctorate  of  Philosophy 
PDE  would  be  the  use  of  probability  distributions  to  complete  the  data  within  column  ten 
(10)  of  Table  5  labeled  L8  Column  Assignments  and  Results  of  Expert  Judge  Group  #1. 

In  this  manor  the  nonintrusive  nature  of  the  research  could  be  maintained  while  providing 
statistically  important  refinement  to  the  linear  equation  result  (Table  7)  predicting  the  WL 
technology  transfer  investment  from  future  R&  D  programs.  Overall  validation  of  the 
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technology  transfer  risk  assessed  investment  linear  equation  prediction  methodology 
would  be  appropriate  for  student  reserach  projects  leading  to  several  master  thesis 
investigating  predicitons  for  each  of  the  WL  focused  outreach  programs  i.e,  automotive, 
medical,  agriculture,  public  safety,  in  addtion  to  the  general  aviation.  Such  a  reserach 
activity  could  conceivably  extend  for  a  period  of  three  to  five  years  to  yield  comparative 
predictions  resulting  from  additonal  expert  judge  panels.  The  WL  metric  leverage  factor 
denominator  term  would  similarly  become  more  vaild  with  additional  years  of  data  and 
experience.  The  combination  of  the  management  tools  into  this  new  methodology 
demonstrates  an  interdisciplinary  nature  in  that  human  factors  and  statistical  processes 
combine  through  the  identified  components  to  produce  a  quantified  measure  predicting 
future  results.  Managers,  with  a  metric  at  hand,  now  have  a  scientific  method  for  decision 
making  about  future  program  investments. 
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9.0  Appendices 


9. 1  Example  CRDA  Benefits  Data 


CRDA  Tracking  Number :  95-201-WL-01 


Company  Involved  :  Paragon  Aircraft 
Point  of  Contact/Office  Svm/Phonc  # : 

Purpose:  Analyze  and  develop  the  design  of  a  single  turbo-fan  engine  aviation  aircraft 
which  has  the  following  attributes: 

-  Inlet  over  the  fuselage,  a  high  aspect  ratio  straight  wing  and  utilizes  a  composite 
structure 

-  Performance  analysis  will  refine  the  aerodynamic  characteristics  of  a  new  air 
flow  and  will  work  to  ensure  the  structural  integrity  of  the  composite  airframe 

Identify  All  the  expected  benefits  received  from  the  CRDA: 

1.  Flight  Dynamics  Branch  gaining  increased  experience  in  CFD  and  Flutter  analysis  and 
ground  vibration  tests 

2.  Usable  test  data  derived  from  the  CRDA-test  conducted  here  at  WPAFB 


3.  Wind  tunnel  testing  still  to  be  conducted  (future  of  the  facilities  may  be  in  jeopardy 
this  CRDA  provides  additional  justification  to  keep  this  facility  operational) 

4.  Royalty  stream  expected  to  recoup  man  hr.  expended  during  this  CRDA. 

Provide  a  quantified  estimate  of  each  benefit  mentioned  above  in  terms  of  dollars 
saved/received,  man  hr.  saved,  or  time  saved. 


CRDA# 

Company : 

Expected  Benefits 
Royalty  Stream 
-Aircraft  Sold 
-Rev.  per  Aircraft 
Payment  for  testing 
*  Fatigue  and  Static 
test  to  be  conducted 
at  WL  facility 
Total  Revenue 
Present  Value  at  7% 


95-201  -WL-01 

Paragon  Aircraft  in  conjunction  with  WL  Flight  Dynamics  Branch 

Net  Present 

1996  1997  1998  1999  2000  2001  Value  at  7%  , 

$  -  $  -  $  $51,622  $98,514  $  37,465 

0  0  07  2.67.1945 1^5.099084  1 .939162 

S  19,320  $  _19,320  $19,320  $'19,320  $19~320'$  197320 
F^$801.417 


$  -  $801,417  $ 
$  -  $699,988  $ 


-  $  51,622  $  98,514 

-  $  39,382  $  70,239 


$  37,465 

$24,964  S  834,574 


Identify  all  of  the  benefits  received  through  CRDA  relationship  that  were  not 
originally  spelled  out  in  the  agreement: 

1 .  The  CRDA  provided  an  opportunity  to  demonstrate  and  validate  existing  capabilities 
at  WL  for  a  whole  airplane 

-  Additional  breadth  gained  by  conducting  developmental  testing  on  an  entire  aircraft 
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2.  The  CRDA  helped  to  justify  the  resources  used  to  do  testing  on  the  aircraft 

3.  Insight  gained  into  the  commercial  aircraft  industry— the  feeling  was  that  WL  had  lost 
touch  with  the  civilian  aircraft  market;  this  CRDA  helped  us  get  our  foot  in  the  door  and 
gain  additional  insight  into  the  civilian  aircraft  industry 

4.  Fatigue  and  static  test  to  be  conducted  at  WL  structures  test  facility— Facility 
contractor  will  receive  payment  for  testing  conducted  at  WL  as  well  as  additional  work 
that  will  help  to  smooth  out  the  testing  schedule.  Additional  knowledge  will  be  gained  in 
the  area  of  flutter  analysis 

*  Paragon  will  pay  the  total  costs  of  static  testing  to  the  contractor  that  runs  the 
structure  test  facility. 


Identify  any  nonquantifiable  benefits  received  by  the  Air  Force  as  a  result  of  the 
CRDA: 

-  Public  relations  benefit-goodwill  towards  the  military  and  Air  Force  in  general 

-  Provides  insight  into  the  FAA  certification  process— this  will  provide  WL  the 
opportunity  to  see  another  approach  to  solving  problems 

-  Increased  capability  at  lower  costs 

Identify  any  lessons  learned  about  the  CRDA  process  during  the  development  of  this 
CRDA 

-  Mutually  beneficial  agreement  where  both  sides  pursued  the  agreement  and  are  actually 
working  to  achieve  a  desired  outcome. 


Crystal  Ball  Report 
Simulation  started  on  7/16/96 
at  15:32:38 

Simulation  stopped  on  7/1 6/96 
at  15:33:02 


Forecast:  Present  value  of  Paragon  CRDA 
Summary 


Display  Range  is  from  $550,000  to  $1,100,000  Dollars 
Entire  Range  is  from  $523,003  to  $1,076,825  Dollars 
After  1,000  Trials,  the  Std.  Error  of  the  Mean  is  $3,237 


Statistics 


./  ..Value 
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Trial 

s 

Mea 

n 

Median  (approx.) 
Mode  (approx.) 
Standard 
Deviation 
Variance 
Skewness 
Kurtosis 
Coeff.  of 
Variability 
Range  Minimum 
Range  Maximum 
Range  Width 
Mean  Std.  Error 


1000 

$834,574 

$843,481 

$935,600 

$102,354 

$10,476,257,251 

-0.35 

2.60 

0.12 

$523,003 

$1,076,825 

$553,822 

$3,236.70 


Forecast:  Present  value  of  Paragon  CRDA  (cont'd) 


Percentiles: 


Percentile 

Dollars  (approx.) 

0% 

$523,003 

10% 

$690,380 

20% 

$740,839 

30% 

$786,464 

40% 

$814,682 

50% 

$843,481 

60% 

$873,552 

70% 

$896,525 

80% 

$928,348 

90% 

$958,340 

100% 

$1,076,82?. 
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Assumption:  Aircraft  sold  year  4 


Triangular  distribution  with  parameters: 


Minimum 

1.00 

Likeliest 

3.00 

Maximum 

4.00 

Selected  range  is  from  1.00 to  4.00 
Mean  value  in  simulation  was  267 


Correlated  with: 

Aircraftsoldyear5  (F7)  025 

Aircraftsold  in  year  6  (G7)  0.15 


Assumption:  Aircraftsold  year  5 


Normal  distribution  with  parameters: 

Mean  5.00 

Standard  Dev.  200 

Selected  range  is  from -Infinity  to  infinity 
Mean  value  in  simulation  was  5.10 

Correlated  with: 

Aircraftsold  year4  (E7) 

Aircraftsold  in  year  6  (G7) 


025 

025 


Assumption:  Aircraftsoldinyear6 


Normal  distribution  vwth  parameters: 

Mean  200 

Standard  Dev.  1.00 

Selected  range  is  from  -Infinity  to  -Hnfinity 
Mean  value  in  simulation  was  124 


Correlated  with: 

Aircraftsold  year  4  (E7)  0.15 

Aircraftsold  year  5  (F7)  025 
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Assumption:  payment  for  static  and  fatigue  tests 


Triangular  distribution  with  parameters: 
Minimum  $500,000 

Likeliest  $900,000 

Maximum  $1,000,000 


Selected  range  is  from  $500,000 to  $1,000,000 
Mean  value  in  simulation  was  $801,417 


paymrtt  tar  aMic  aita  taOgtM 


•MO.OOO  MH.0M  HIMOO  IUOIOOO 
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9.2  Outreach  to  General  Aviation  Task  Plan 
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U.S.  AIR  FORCE  WRIGHT  LABORATORY 
PLANS  &  PROGRAM  DIRECTORATE 
TECHNOLOGY  TRANSFER  &  TRANSITION 
STRATEGIC  PLAN 


Task  Plan:  Outreach  to  General  Aviation 

Relevant  Action  Plan:  Expand  Outreach  Programs 

Focal  Point:  Mr.  Lanny  A.  Jines  P.E.  (jinesla@wl.wpafb.af.mil) 

Objective:  Assist  the  U.  S.  General  Aviation  Industry  to  achieve  World  Market 
Leadership  in  the  design,  development,  and  manufacturing  of  all  category  and  class  of 
aircraft  through  technology  transfer  and  transition  while  insuring  military  readiness  via 
sustained  availability  of  advanced  aerospace  technology  production  capability  and 
capacity. 

Assumptions:  The  General  Aviation  Revitalization  Act  (S.  1458)  of  1994  signed  by  The 
President  reform  federal  tort  law  establishes  an  eighteen  (18)  year  statue  of  repose  in 
aircraft  accident  lawsuits.  This  precedent  setting  legislation  provides  the  general  aviation 
industry  new  opportunities  for  original  equipment  manufacturers  (OEM),  after  market 
vendors,  modification  facility  operators,  and  material  suppliers  to  design,  develop,  test, 
and  market  advanced  state-of-the-art  technology  improvements  in  materials, 
manufacturing  processes  and  aircraft  designs  for  flight,  ground,  and  training  systems. 
Wright  Laboratory's  technologies  combined  with  the  design,  production,  and  market  share 
capabilities  of  the  General  Aviation  Industry  would  reduce  development  and  life  cycle 
costs  of  militarily  important  advancements. 

Approach:  Market  WL  technology  transfer  and  transition  opportunities  through  the 
Advanced  General  Aviation  Transport  Experiments  (AGATE)  Program.  This  program  is 
a  consortium  composed  of  NASA,  FAA,  and  General  Aviation  Industry.  Additionally, 
establish  a  Technology  Transfer  Outreach  to  General  Aviation  category  within  the  Internet 
World  Wide  Web  (WWW)  WL  Home  Page .  Post  and  maintain  on  the  WWW  current  list 
and  description  of  applicable  points  of  contact  and  technologies  ready  for  transfer  to 
general  aviation.  Focus  and  control  outreach  opportunity  to  capitalize  upon  U.S.  Air  & 
Trade  Show  for  WL  technology  and  facility  marketing  and  exhibition. 

Air  Force  Relevance:  The  advanced  technologies  in  materials,  structures,  propulsion, 
flight  control,  avionics,  cockpit,  subsystems,  and  training  are  common  to  all  aircraft 
systems,  military  and  civilian  general  aviation  alike,  seeking  to  meet  customer  need  for 
improved  safety,  performance,  and  efficiency.  The  timely  and  cost  effective  availability  of 
materials,  manufacturing  facilities,  skilled  work  force  and  proven  systems  integrated 


85 


performance  important  to  Air  Force  weapon  systems  (i.e.  commercial  available  products 
for  direct  military  use)  is  directly  related  to  the  proliferation  of  aircraft  design,  production, 
operations,  and  training  systems  in  a  modem  state-of-the-art  United  States  manufactured 
general  aviation  fleet  competitive  in  the  world  market. 

Payoff:  The  General  Aviation  Industry  should  benefit  by  having  access  to  WL 
technologies  and  resources  while  the  Air  Force  shall  have  access  to  "smarter,  cheaper, 
better"  aircraft  flight  systems,  propulsion  sensors  and  controls,  integrated  design  and 
manufacturing,  icing  protection  systems,  integration  platforms  for  flight  testing,  air 
infrastructure  strategies,  ground  infrastructure  improvements,  advanced  technology 
training  systems,  and  aerodynamic  acoustic  improvements  for  future  military  systems. 
Additional  payoff  to  the  Air  Force  includes  the  access  to  stronger  more  competitive 
domestic  manufacturing  industrial  base  with  a  work  force  of  compatible  and  relevant  skills 
available  when  military  needs  arise. 

Schedule: 


Major  Activity 

Start 

Complete 

Preparation/Planning 

02  JAN  96 

28  FEB  96 

Core  PAT 

01  MAR  96 

30  MAR  96 

WL  Technology  Identification  Process 

01 -APR  96 

ON-GOING 

GA  Technology  Need  Identification  Process 
(FAA/NASA/  GA  Industry  T3  Visits) 

01  APR  96 

ON-GOING 

General  Aviation  (GA)  Outreach  Home  Page 

01  MAY  96 

30  MAY  96 

SBIR  &  CRD  A  Development 

01  JUN96 

ON-GOING 

GA  Outreach  Functional  IOC 

01  JUN  96 

ON-GOING 

AGATE  Membership 

31  JUL  96 

ON-GOING 
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9.3  CRDA  Ballot  Data 


Company 


CRDANo. 


Night  Vision  Eqpt  PA 

92-281-1W 

Lockheed  FW  TX 

93-126- wl-01 

Superconductive  Comp  OH 

94-178-W1-02 

Shaw  Custom  Sys's  CA 

94-241 -wl-01 

Loral  Defense  System  OH 

95-009-wl-01 

Am'cn  CompTech  OH 

95-332-wl-01 

Adtech  Sys  Res  Lab  OH 

96-101-wl-01 

Gauge  &  Measure  OH 

94-010-wl-01 

Northrop  Elect  Sys  IL 

94-178-wl-01 

VPI 

92-175-IAF 

Firefox  Industries  PA 

94-173-wl-01 

Artificial  Horizons  CA 

94-173-wl-02 

Rockwell  Int'l  Corp  CA 

94-242-wl-02 

The  Main  Corporation  CO 

94-272-wl-01 

Smiths  Industries  FL 

94-327-wl-01 

Cold  Jet  OH 

94-336-wl-01 

Aviation  Envi'l  Comp  OH 

95-075-wl-01 

Fatigue  Concepts  CA 

95-075-wl-02 

Lear  Astronics  Corp  CA 

95-093-wl-01 

Metrolaser  CA 

95-170-wl-01 

Adtech  Systems  OH 

95-192-wl-01 

Universal  Analytics  CA 

95-192-wl-02 

Century  Aerospace  NM 

95-201 -wl-01 

Intem'l  Mktg  Inc  PA 

95-214-wl-01 

Systran  Corp  ’  OH 

95-222-wl-01 

88 


Hughes  Danbury  CT 


Textron  Spec'y  Matls  MA 


Hitco  Techn'ies  Inc  CA 


Rapco  Fleet  Serv's  WI 
Univ  of  Missouri  MO 


Adtech  Sys  Rsrch  OH 


Adtech  Systems  OH 


Daychem  OH 


Purdue  University  IN 


Adtech  Systems  OH 


Performance  Plastics  OH 


Process  Eqpt  Co.  OH 


Ribbon  Technology  OH 
General  Electric  NY 
General  Electric  NY 
Technosoft  Corporati  OH 
Armco  Inc.  OH 


Techy  Assmt  &  Trans  MD 
Atlantic  Research  MA 


Modern  Techn'ies  OH 


Edison  Polymer  Inc  OH 


Applied  Science  Inc  OH 
U.  of  Rochester.  NY 


Hohman  Plat'g/Mfg  OH 


Laser  Photonics  NY 


95-319-wl-01 


96-005-wl-01 


96-130-wl-01 


96-134-wl-01 

96-222-wl-01 


96-323-WL-01 


89-01 


90-01 _ 


91-01 


91-02 


93-099-wl-01 

93-207-wl-01 

93-250-wl-Ql 

93-267-wl-02 

93- 35  l-wl-01 

94- 012-wl-Ql 
94-130-wl-Ol 


94-132-wl-01 

94-171-wl-Ol 


94-242-wl-01 


94-340-wl-01 


95-004-wl-01 

95-156-wl-Ol 


95-335-wl-Ol 


96-030-wl-Ol 
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Century  Aerospace  NM 

96-149-wl-01  ] 

Maxdem  Inc.  CA 

96-156-wl-01 

TCAM  Tch'ies  Inc.  OH 

96-27  l-WL-01 

RJO  OH 

93-267-wl-01 

GE  Acft  Engines  OH 

93-208-wl-01 

Davis  Engineering  TN 

93-22  l-wl-01 

Allied  Signal  Contro  IN 

94-03 8-wl-01 

Solar  Turbines  Inc.  CA 

96-078-wl-01 

GE  Acft  Engines  OH 

96-145-wl-01 
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9.4  ASTARS  Data  for  Matching  WL  R&D  Technologies 


JON 

Begin 

End 

R&D  $K 

T2  Potential 

43472403 

1 -Mar-96 

31 -Dec-97 

135 

HI 

300505Q6 

1 -Mar-95 

1 -Mar-97 

275 

HI 

43471102 

l-Jun-96 

30-Sep-00 

1082 

HI 

300505R6 

1 -Feb-96 

31 -Dec-99 

643 

HI 

43476107 

l-Jun-95 

30-Sep-98 

300 

LO 

2302P101 

1 -Oct-84 

30-Sep-99 

1558 

LO 

31530015 

1 -Aug-94 

31 -Dec-00 

2837 

LO 

31530016 

l-Aug-94 

31 -Dec-97 

951 

HI 

43494102 

31 -Jan-96 

30-Jun-98 

35 

LO 

4349S101 

1 -Sep-96 

30- Apr-03 

1355 

HI 

43494401 

3-Jan-95 

30-Sep-98 

676 

HI 

24180260 

1 -Oct-92 

30-Sep-99 

896 

LO 

24180485 

10- Aug-92 

30-Oct-97 

1684 

LO 

24184005 

1 -Sep-93 

30- Aug-99  ' 

4339 

HI 

24184008 

l-Apr-93 

31 -Mar-97 

737 

LO 

24184012 

l-Dec-93 

30-Dec-96 

165 

HI 

24184017 

15-Jul-94 

30-Jun-97 

193 

HI 

30050581 

14-Jun-93 

30-Mar-98 

822 

LO 

43494001 

l-Sep-96 

30-May-03 

4992 

HI 

300505 AB 

15-Mar-94 

31 -Dec-97 

738 

HI 

24803000 

13 -May-92 

31 -Jan-98 

2075 

LO 

24803050 

24-Nov-93 

30-Dec-99 

1630 

LO 

24803051 

1 -May-94 

30-Dec-99 

560 

LO 

24803052 

1 -May-94 

30-Dec-99 

157 

LO 

24803054 

1 -May-94 

30-Dec-99 

893 

LO 
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30480537 

1 -Oct-83 

3 04805 AJ 

1 -Oct-92 

314501C6 

5-Nov-91 

314501C8 

2-Jun-93 

3 14520C3 

13-May-93 

314529LA 

19-May-93 

314532Y4 

l-Jun-93 

2303BW1Q 

l-Oct-95 

30480618 

1 -Apr-72 

30950126 

l-Sep-93 

30950530 

28-Feb-93 

30950113 

5-May-91 

CRDF9507 

1 -Oct-95 

30054192 

15-Aug-95 

30054191 

15-Aug-95 

ARPAAA09 

25-Aug-95 

76620166 

3 -Jan-94 

25060701 

1 -Nov-89 

24030159 

13-Apr-92 

240301 5 J 

l-Jul-94 

2403015U 

1 -Feb-96 

2403015V 

21 -Mar-96 

30O5F0O0 

29-Dec-95 

2302BW2P 

1 -Oct-95 

30-Sep-97 


30-Mar-98 


30-Sep-97 


31 -Aug-98 


30-Sep-97 


30-Sep-99 


1 -Sep-99 


31 -Aug-99 


30-Sep-97 


30-Sep-97 


1 -Oct-99 


30-Sep-98 


30-Sep-98 


26-Feb-98 


31 -Dec-99 


30-Sep-97 


30-Dec-97 


30-Sep-98 


l-Jun-97 


30-Mar-97 


15-Feb-98 


30-Sep-97 


31 -Dec-97 


1350 


2013 


2954 


674 


652 


521 


576 


105 


1093 


3175 


2000 


23424 


300505 AB 


15-Mar-94 


738 


9.5  Probability  Encoding  Questionnaire 


Subject:  Assessment  of  Wright  Laboratory  R&D  Program  Potential  Technology  Transfer 

Focused  out  reach  efforts  are  identified  in  the  Wright  Laboratory  Technology  Transfer  and  Transition 
Strategy  Plan.  They  exist  to  assist  the  U.S.  industrial  manufacturing  base  sustainment  of  world  market 
leadership.  The  task  plan  for  the  Out  Reach  to  General  Aviation  resulted  from  theWright  Laboratory  strategy. 
Through  your  assistance,  the  Wright  Laboratory  Directorates  recently  reviewed  their  current  R&D  technology 
programs  which  exhibited  strong  relationship  to  the  operational  requirements  of  the  NASA  -  Industry  -  FAA 
Advanced  General  Aviation  Transport  Experiments  (AGATE)  program  for  revitalization  of  the  general 


R&D  Program  ASTARS  JON: _ _ 

1 .  In  your  expert  opinion  does  the  R&D  program  (See  ASTARS  JON  Data)  exhibit  potential  for  transfer  of 
technology  to  an  industry  partner  resulting  in  the  generation  of  revenue  via  a  CRDA  fee.  Patent  fee,  or  License 
fee. 


aviation  industry.  The  response  of  the  directorates  resulted  in  a  listing  ot  technology  programs,  iacinties,  i 
expertise  which  would  represent  the  Wright  Laboratory  capability  for  technology  transfer  assistance  to  the 
general  aviation  industry.  The  ASTARS  database  contains  information  about  the  identified  programs. 


A.  Yes  B.  No  (If  No  then  skip  remaining  questions  proceed  to  next  ASTARS  report) 

2.  In  your  expert  opinion  which  level  of  Air  Force  investment  would  be  required  to  bring  about  the  transfer  of 
technology  from  the  R&D  program  under  consideration.  The  levels  are  stated  as  a  percentage  or  decimal 
factor  of  the  R&D  program  total  cost  as  shown  in  the  ASTARS  report.  Note:  The  median  Wright 
Laboratory  investment  per  General  Aviation  Out  Reach  CRDA  for  the  technology  tranfer  efforts  of  Air  Force 
unit  scientist  and  engineers  ha  s  been  determined  to  be  $  6,045. 

A.  0.05%  or  .0005  B.  0.10%  or  .001  C.  1.0%  or  .01 

3.  In  your  expert  opinion  which  of  the  following  schedule  periods  would  most  likely  allow  for  anticipated 
R&p  program  technology  transfer  to  an  industry  partner.  (ASTARS  progress  reports  may  be  useful) 

A.  Six  (6)  Months  B.  Eighteen  (18)  Months  C.  Thirty-six  (36)  Months  or  longer 

4.  In  your  expert  opinion  please  assign  a  technology  transfer  risk  assessment  value  to  the  R&D  program. 
(Note:  The  ASTARS  report  in  some  cases  has  an  input  for  Technology  Transfer  potential  provided  by  the 
Project  Engineer.  You  may  generally  agree  or  disagree  with  theASTARS  information) 

A.  0.01  One  or  more  CRDAs  already  exist  based  upon  this  technology  R&D  program. 

B.  0.2  The  R&D  program  has  previously  produce  technology  transfer  activities  which  have  been 

completed  (CRDA  fulfilled  or  expert  assistance  provided)  but  some  effort  required  for  new 
applications. 

C.  0.5  At  the  end  of  the  scheduled  period  noted  in  question  three  (3),  the  technology  transfer 

activity  could  produce  a  technology  for  manufacture  or  use  in  production  processes. 

D.  0.7  The  R&D  program  is  a  proof  of  concept  Reasonable  effort  would  be  required  to  bring 

about  transfer  of  technology  into  new  product  or  manufacturing  process. 

E.  0.9  The  R&D  program  is  basic  research.  Significant  effort  would  be  required  to  bring  about 

transfer  of  technology  into  new  product  or  manufacturing  process. 
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9.6  Questionnaire  Results 


JON 

R&D  Prgm  $  K 

24180260 

896 

43472403 

135 

300505Q6 

275 

43471102 

1.082 

300505R6 

643 

43476107 

300 

2302P101 

1,558 

31530015 

2,837 

31530016 

951 

43494102 

35 

4349S101 

1,355 

43494401 

676 

24180485 

1,684 

24184005 

4,339 

24184008 

737 

24184012 

165 

24184017 

193 

30050581 

822 

43494001 

4,992 

300505AB 

738 

24803000 

2,075 

24803050 

1,630 

24803051 

560 

24803052 

157 

24803054 

r  893 

30480537 

1,350 

304805AJ 

2,013 

314501 C6 

2,954 

31 4501 C8 

674 

314520C3 

652 

314529LA 

521 

314532Y4 

576 

2303BW1Q 

105 

30480618 

1,093 

30950126 

3,175 

30950530 

2,000 

30950113 

76620166 

841 

24030159 

4,906 

2302BW2P 

V  o 

CRDF9507 

7 

25060701 

3.317 

T2  Invstmnt  % 


NA 


0.05% 


0.05% 


0.05% 


0.05% 


0.05% 


0.05% 


0.05% 


0.05% 


0.05% 


0.05% 


0.05% 


0.05% 


0.05% 


0.05% 


0.05% 


0.05% 


0.05% 


0.05% 


0.05% 


0.05% 


0.05% 


0.05% 


0.05% 


T2  Mnths 


NA 


6 


6 


6 


6 


6 


6 


6 


6 


6 


6 


6 


6 


6 


6 


6 


6 


6 


6 


6 


6 


6 


’  6 


6 


6 


6 


6 


6 


6 


6 


T2  Risk 


NA 


0.01 


0.01 


0.01 


0.01 


0.01 


0.01 


0.01 


0.01 


0.01 


0.01 


0.01 


0.01 


0.01 


0.01 


0.01 


0.01 


0.01 


0.01 


0.01 


0.01 


0.01 


0.01 


0.01 


0.01 


0.01 


0.01 


0.01 


0.01 


0.01 


0.01 


0.01 


0.01 


0.01 


0.01 


0.01 


0.01 


0.01 


0.01 


0.01 


0.01 


T2  Estmtd  $K 


#VALUE! 


0.068175 


0.138875 


0.54641 


0.324715 


0.1515 


0.78679 


1.432685 


0.480255 


0.017675 


0.684275 


0.34138 


0.85042 


2.191195 


0.372185 


0.083325 


0.097465 


0.41511 


2.52096 


0.37269 


1.047875 


0.82315 


0.2828 


0.079285 


0.450965 


0.68175 


1.016565 


1.49177 


0.34037 


0.32926 


0.263105 


0.29088 


0.053025 


0.551965 


1.603375 


1.01 


11.82912 


0.424705 


2.47753 


0.0707 


56.389 
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30054192 

555 

1 

1.00% 

36 

wm 

9.435 

30054191 

450 

1 

1.00% 

36 

0.7 

7.65 

240301 5J 

18 

1 

1.00% 

36 

0.7 

0.2975 

240301 5U 

300 

1 

1.00% 

36 

0.7 

5.1 

2403015V 

225 

1 

1.00% 

36 

0.7 

3.825 

ARPAAA09 

754 

1 

1.00% 

36 

0.9 

14.326 

3005F000 

750 

1 

1.00% 

36 

0.9 

14.25 

